Complex computing systems are nowadays oftentimes implemented based on the so-called “event-driven architecture” (EDA), which is a software architecture pattern promoting the production, detection, consumption of and reaction to events. Such systems are also called “Event Processing Networks” (EPN) (cf. “Event Processing Network—A Conceptual Model” of Sharon et al., VLDB'07, Sep. 23-28, 2007, Vienna, Austria, and “Event Processing Under Uncertainty” of Artikis et al., DEBS'12, Jul. 16-20, 2012, Berlin, Germany). The EDA pattern may be applied in the implementation of applications and systems which transmit events among loosely coupled software components and/or services. The terminology used in the present application adheres to the terms used in the “Event Processing Glossary—Version 1.1” by the Event Processing Technical Society (Editors: David Luckham, Roy Schulte, July 2008).
An event processing network (EPN) describes the event processing flow execution and is generally illustrated in FIG. 1 in a simplified form. The exemplary EPN comprises event producers 10 (or event sources 10), one or more event channels 30 forming an event bus, and event processing agents 40 (EPAs). The network might further comprise one or more event consumers (not shown in FIG. 1).
Generally, event processing typically follows the pattern of continuous queries, i.e. queries that execute for an infinite time on always changing data (so called event streams), comprising events. Such event streams, often exist in real-world scenarios, e.g. temperature readings from sensors placed in warehouses or trucks, weather data, entrance control systems (where events are generated whenever a person enters or leaves) etc. The input events comprise attributes such as values of temperature readings (also called payload) and metadata e.g. a creation date, a validity period, and a quality of the event (typically comprised in a header of the event). As shown in FIG. 1, event streams, comprising input events, are derived from respective event sources 10. The event channel 30 (or event channels) provides an interface between the event sources 10 and the EPAs 40. The event channel 30 is typically implemented by an event bus. The event channel 30 organizes and combines the event streams for further transmission to the EPAs 40. The EPAs 40 are also known as event mediators and are in charge of tasks such as pattern detection, processing of selected event streams satisfying the pattern, and generating and emitting derived events or streams of events. Typically, all possible events occurring in an EPN are schematically described by event types (comparable to table definitions in relational databases). Event types can e.g. represent raw events and/or derived events. Channels 30 and event types are orthogonal concepts: channels may comprise events of several event types, events of the same event type might be communicated via different channels 30. The events might be represented in XML format. For example, an event originating from a temperature sensor reading in a moving vehicle (e.g. a truck transporting temperature-sensitive goods) could be represented in the following manner:
<TempReadingxmlns=”http://softwareag.com/eventtypes/temperaturereading”><header valid-from=”20130822:10:59:00” valid-to=”20130822:11:00:00”/><payload><truckId>59834</truckId><temperature>7.6</temperature><location><lat>48.200000</lat><lon>16.260000</lon></location> </payload></TempReading >
Consumers (not shown in FIG. 1) have the responsibility of applying a reaction as soon as an event they consider relevant is presented. In summary, building applications and systems around an EDA allows these applications and systems to be constructed in a manner that facilitates more responsiveness, because event-driven systems are, by design, more normalized to unpredictable and asynchronous environments.
Complex event processing (CEP) is a technique of processing events to create complex events, wherein a complex event is an event that is an abstraction (aggregating, summarizing, representing, denoting) of events or sets of events that fit a pattern. In CEP, the queries are comparatively fixed and fed with continuously arriving streams of data (input events or raw events). CEP queries (also referred to as “continuous queries”) typically correlate multiple input data items, look for patterns and produce output events (or derived events) when a certain pattern is observed. The CEP queries are often formulated in a SQL-like query language, which is enhanced by CEP specific clauses, such as windows or range clause to define conditions and are typically executed by said EPAs.
In contrast to database systems which run queries once on a certain state of the data, CEP systems do continuous query execution on streams, i.e. a query is essentially evaluated “forever”. This allows CEP systems to spend much more effort on query optimization, as query compilation occurs only once unless the query is modified. On the other hand, CEP systems need a mechanism for “hot redeployment” of queries to cope with changes in queries.
In summary, CEP applications have to deal with highly transient event data that arrives continuously at very high rates and have to produce the corresponding output events/alerts as soon as possible, ideally nearly in real-time. This includes push-based processing (also known as data driven processing) preferably within main memory, which denotes a data flow approach where data to process is not requested (pulled) by the processing operator on demand or using certain scheduling techniques, but is directly provided (pushed) to the processing operator when it becomes available. The U.S. Pat. No. 7,676,461 B2 and U.S. Pat. No. 7,457,728 B2 provide further background information about complex event processing (CEP).
Further, it is known that event processing has to deal with huge amounts of data which has to be processed, referred to as “big data”. The term “big data” is characterized by several factors. The factor veracity (“data in doubt”) is becoming increasingly important. Veracity is defined as the uncertainty in data. In other words, the input event streams carry inherent uncertainties, such as incomplete or inaccurate data streams. For example, a temperature sensor continuously determining temperature readings may be damaged by certain circumstances, where this might result in wrong temperature readings and therefore in erroneous values. Hence, the input events produced by this device which comprise the respective values are incorrect as well. In this context, those skilled in the art will appreciate that such handling of inherent uncertainties and/or erroneous or faulty data is a difficult task given the huge amount of data to be processed in a correct and efficient manner.
For example, consider an event-based system that operates a control system in an aircraft for controlling distinct operations, such as the landing of the aircraft. The control system might continuously receive a large amount of input events, each comprising a plurality of current parameters or attributes, such as GPS coordinates, height of the aircraft and speed of the aircraft. The control system is programmed to start a controlled landing if the parameters reach a predetermined threshold. This might be particularly achieved by applying a specific landing configuration. If the configuration comprises incorrect events and parameters, this might lead to catastrophic consequences. For example, the aircraft may touch the ground too early, with high speed beyond the landing strip, thereby resulting in a plane crash in the worst case.
Therefore, various approaches have been proposed in the prior art for dealing with inherent uncertainty in event processing. The prior art document “Event Processing Under Uncertainty” of Artikis et al. (DEBS'12, Jul. 16-20, 2012, Berlin, Germany) provides a comprehensive overview of types of uncertainty, uncertainty representation and uncertainty handling for event processing systems. In particular, two uncertainty handling methods have been proposed: uncertainty propagation and uncertainty elimination, as will be explained in the following. In uncertainty propagation, the uncertainty/uncertainties in input events is/are propagated to the derived events. However, the first prior art solution has the disadvantage that the uncertainty has to be handled in the expressions which are commonly defined in event processing, particularly in the queries of the EPAs. The expressions comprise for example arithmetic functions and logical operators. The first prior art approach particularly suggests to adapt the defined expressions in a complex manner. For example, the expressions are overloaded or extended to accommodate stochastic values, such that stochastic expressions would evaluate to a distribution over a certain domain.
Contrary, elimination of uncertainty suggests removing the uncertainty before derivation, for example by replacing uncertain attributes by deterministic equivalents and screening out uncertain events according to predefined policies. The second prior art solution suggests in particular using probabilistic functions in the expression of the EPAs.
This elimination approach has the disadvantage of losing crucial information. It is well reported that processing of specific and reduced subsets of data leads to a reduced accuracy and completeness of results.
However, both prior art approaches achieve the uncertainty handling in the EPAs themselves, since error correction and in particular the complex adaptations to expression/queries are defined in the EPAs. In other words, each EPA has to be aware of the various types of errors that might occur and has to provide respective error handling measures.
Other approaches do not adapt the expressions in the EPAs at all. The inherent uncertainties of event streams are irrespectively incorporated in event processing or derivation, resulting in uncertain derived events. Since the whole set of input events, comprising correct as well as incorrect input events, are subject to processing, the EPAs cannot perform reliably and efficiently. Complex and time-consuming prior art approaches, for example the U.S. patent application publications 2009/0125550 A1 and 2013/0031567, are directed to a technique called “retraction” to deal with such uncertain derived event data. Retraction can be considered as a way of performing speculative execution. In the case that output or derived event streams turn out to be incorrect, the processor (in particular the EPA) can retract the incorrect event data and resend the correct event data. This way, the event processing system is slowed down by the process of retraction since the retracted event data has to be retreated and processed again.
In other words, the developer of each event based application has to be aware of uncertainties and possible errors and has to consider these in his event stream processing programs/queries, which is a considerable burden and prohibits fast and slim event processing application development. In addition, erroneous events could lead to system problems (e.g. a CEP system might cease operation if it encounters out-of-time-sequence events).
It is therefore the technical problem to provide an event-based correcting system which removes incorrect event data without losing crucial information and without complex adaptations to a given CEP system, but at the same time performing in an efficient and scalable manner, thereby at least partly overcoming the above explained disadvantages of the prior art.
This problem is according to one aspect solved by a complex event processing, CEP, system. In the embodiment of claim 1, the CEP system (1) comprises:
a. an error correction component (20), adapted for receiving a stream of events comprising at least one event from at least one event source (10);
b. wherein the error correction component (20) is adapted for detecting at least one error in the at least one event; and
c. wherein the error correction component (20) is adapted for emitting a corrected stream of events comprising at least one event, which can then be processed by at least one event processing application (40).
Accordingly, the embodiment defines a CEP system for detecting and correcting erroneous input events in event streams (wherein a stream of events is defined as a sequence of zero or more events), in particular in “big data”. To this end, the CEP system comprises an error correction component which is decoupled from the prior art-EPAs. Further, the implementation and/or configuration of the error correction component might be also decoupled from the prior art-EPA queries, as will be explained further below. Importantly, the error correction component consumes input events (comprised in a stream of input events) which are derived from the event sources, and is capable of detecting various error types in the input events. For example, an input event might be completely missing in the continuous sequence of input events or an error in an input event might have occurred, as will be explained further below. If an error is detected by the error correction component, the error is handled directly by the error correction component itself (“self-correcting mechanism”). This way, a corrected stream of events is produced by the error correction component, which is then emitted for further processing. Particularly, the corrected event stream can serve as input for event processing applications, such as EPAs.
As it is known in the prior art, the uncertainty handling and respective adaptations to the expressions are commonly defined in the EPAs. In other words, the EPAs in the prior art receive both, correct and incorrect, input events. Hence, the EPAs have to deal with uncertainty handling as well as processing of a huge amount of erroneous data. To the contrary, certain example embodiments allow for decoupling the task of error detection and correction from the EPAs. Particularly, the erroneous events are no longer transmitted to the EPAs since these events are either corrected or excluded before they reach the EPAs, as will be explained further below. This way, the data load and work load is reduced in the EPAs allowing for a reliable and efficient operation by the EPAs.
Although the error correction component might be implemented itself as an EPA, certain example embodiments differ from the prior art approaches in that the error correction component is able to correct errors in input events “upstream” of the actual event processing performed by the EPAs (and preferably even before the events are placed on the event bus of the system). In an embodiment, the error correction component can be connected to an event channel and/or the event processing application which is preferably implemented as at least one second EPA. More specifically, in a preferred embodiment, the error correction component receives input events from the event sources, performs the error correction and transmits the corrected stream of events to an event bus comprising one or more event channels and therefore to at least one second EPA only via the event bus. The event channel(s) e.g. might be a common event bus, facilitating the communication of corrected events between the error correction component and the at least one second EPA. The at least one second EPA preferably differs from the error correction component in that it does not have to bother with the error correction, thereby avoiding the prior art adaptations to queries/expression. In other words, the at least one second EPA is adapted for event processing or any other operations except for error correction or uncertainty handling, such as common prior art-EPAs. Contrary to the prior art-EPAs, the error correction component may be capable of handling distinct event types and error types, such as out-of-sequence events. However, the error correction component does not have to implement and/or understand the query logic of prior art-EPAs.
In one aspect, the error correction component is adapted for detecting the at least one error in the at least one event based on at least one configuration file specifying an expected behavior of the at least one event source, comprising an indication whether time-ordered event delivery is expected, a maximum tolerated event delay time and/or an expected arrival rate of events. Accordingly, the event sources generate input events in a continuous and predictable manner. These input events are expected to be delivered from the event sources and to arrive in accordance with the expected behavior of the event sources. In particular, the input events can be e.g. ordered by time stamps and are expected to be delivered and to arrive in the respective time-ordering. Additionally or alternatively the input events might be delivered with a maximum tolerated delay time and/or arrival rate. This expected behavior of event sources is preferably defined in a configuration file. If the event source does not deliver the expected behavior, the error correction component is capable to generate this expected behavior. This has the advantage that the types of errors to be detected can be defined and managed in a central place, namely the configuration file, in contrast to being scattered among the EPAs, as done in the prior art.
Furthermore, using central configuration file(s) has the further advantage that the error correction can be flexibly adapted to changed circumstances by simply editing the configuration file(s).
In a further aspect, the at least one configuration file further specifies at least one error correction parameter for the at least one event, comprising e.g. an allowed value range, an allowed change rate and/or an expected value distribution. Accordingly, the configuration file further comprises error correction parameters for the input events. For example, a temperature sensor as one example of an event source may determine temperature readings. The temperature is measured in degrees and is e.g. known to have a certain range of degrees. These determined values are essential for error corrections and are comprised as error correction parameters in the configuration file for the input events.
In a further aspect, the at least one configuration file further specifies at least one error correction method to be applied by the error correction component, comprising a reference to a function for calculating at least one corrected event, ignoring at least one event, replacing at least one erroneous value of at least one event with at least one value of a previous event and/or generating at least one interpolated value for at least one event. Accordingly, the configuration file can be used not only for detecting errors and abnormal behavior in the input events by the error correction component, but the configuration file might further comprise instructions on how to correct detected errors. The error correction method is applied on (events of) the input event stream(s), if errors are detected in the input event stream(s). Various types of error correction methods are provided. For example, an error in an input event can be corrected and the corrected event can be emitted by the error correction component. In particular, an error may be corrected with a function for calculating a respective corrected event for further processing. Further, the erroneous event can be completely discarded whenever no correction is required or reasonable and is accordingly not maintained in event processing. Instead of discarding an erroneous event, the previous and/or next events can be incorporated for replacement or interpolation.
Preferably, the at least one configuration file is stored in a repository. Accordingly, the configuration file which is essential for detection and correction of erroneous input events is stored in a central place. This way the computing components in the EPN, particularly CEP system, have access to the configuration file. Importantly, the error correction component as computing component is decoupled from the configuration file itself. In the case of adaptations e.g. the expected behavior of event sources has changed, solely the configuration file has to be updated. This way, the error correction component is not affected by or even aware of the adaptations and the overall configuration of an arbitrarily complex CEP system can be managed in a central location.
In yet another aspect, the at least one configuration file specifies a grouping of events, and wherein the error correction component is adapted for applying an error correction method on the grouping of events. Accordingly, the input events are derived from distinct event sources, wherein an event source comprises sensor readings from distinct sensors of a particular type or group (e.g. temperature sensors). Each of the sensors has a unique ID as grouping criterion. The error correction method is applied on the input events of the same group (with the same group ID). This is particularly advantageous since the expected behavior might differ among distinct groups and the input events are corrected respective to their group.
In a further aspect, the error correction component is further adapted for emitting at least one error event indicating that the at least one error in the stream of events was detected, wherein the at least one error event can be consumed by a monitoring component. Accordingly, distinct event types can be emitted by the error correction component. As mentioned above, a corrected event stream is emitted and preferably transmitted to the EPAs for event processing. Additionally or alternatively, error events can be emitted by the error correction component. The error events comprise information about the errors in the input events, irrespective if they were corrected or not. The error events can be used for monitoring, e.g. by using approaches such as disclosed in the European patent application EP 13169119.8 of applicant. The type of the error event and other details can be configured in the configuration file as well.
Preferably, the at least one event and/or the at least one configuration file is XML based.
In yet another aspect, the error correction component is capable of adapting the at least one configuration file based on the corrected stream of events and/or the at least one error event. Accordingly, the distinct events are essential to gain knowledge about the normal and abnormal behavior in the CEP system. In accordance with the generated events, the configuration file can be adapted to changed behavior, thereby improving the error correction and/or monitoring by the error correction component by means of self-learning.
Certain example embodiments relate to a method for error correction in a CEP system in accordance with the any of the above-described aspects. Lastly, the certain example embodiments provide a computer program (e.g., stored on a non-transitory computer readable storage medium) comprising instructions for implementing the above-described method.